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Abstract 

The purpose of this study was to document within-year fidelity growth during installation and 
initial implementation of School-Wide Positive Behavioral Interventions and Supports 
(SWPBIS). Participants included school teams from schools throughout the United States that 
were in years one to four of SWPBIS implementation and routinely evaluated their 
implementation fidelity. The fidelity outcome was assessed with the Team Implementation 
Checklist (TIC) and was completed multiple times per year by SWPBIS teams. Results from 
multilevel fidelity growth models documented within- and between-school variability and 
growth predictors. Years implementing, location, school type and enrollment size were 
significant predictors of beginning year fidelity scores (intercept), and years implementing and 
relative socio-economic status were significant predictors of the average rate of fidelity change 
per month of school (slope). These results add to the growing field of implementation science, 
and practice recommendations are outlined. 
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Within-year Fidelity Growth of SWPBIS during Installation and Initial 

Implementation 

School-Wide Positive Behavioral Interventions and Supports (SWPBIS) is a systems 
intervention framework focused on creating healthy school environments by promoting pro¬ 
social behavior and supporting students with behavioral needs that is implemented in over 
21,000 schools across the United States (Horner, 2014). The program focuses on universal 
behavioral expectations implemented in every classroom across schools, administrative functions 
to facilitate ongoing implementation, and data use to identify both students with behavioral needs 
and areas for programmatic improvement. In multiple effectiveness studies, SWPBIS has been 
documented to decrease office discipline referral and suspension rates, increase on-task academic 
behaviors, and improve academic performances (Algozzine & Algozzine, 2007; Bradshaw, 
Mitchell, & Leaf, 2010; Horner, Sugai, & Anderson, 2010). Documenting impact on valued 
outcomes is one piece of essential evidence for scaling educational innovations, and another is an 
understanding how to effectively put a practice into place. Implementation science provides a 
theoretic lens and practical steps for installing and sustaining research-based innovations within 
real-world organizations that can be monitored with accurate fidelity measures. Given its 
evidence of effectiveness and large-scale implementation, SWPBIS provides an excellent 
opportunity to explore implementation variables related to implementation fidelity. 

Importance of SWPBIS Implementation Fidelity 

Implementation fidelity provides an understanding of the implementation process that is 
useful for ensuring effective program outcomes as fidelity is related to valued school and student 
outcomes. A randomized control trial conducted by Horner and colleagues (2009) documented a 
functional relationship between improved SWPBIS implementation and improved perceptions of 
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school safety, third grade academic achievement, and decreased discipline referral and 
suspension rates. Another randomized trial showed that effective implementation was related to 
improvements on measures of organizational health, staff affiliation and academic emphasis 
(Bradshaw, Koth, Bevans, Ialongo, & Leaf, 2008). A recent study supports this claim by 
showing that increased SWPBIS fidelity resulted in significantly decreased office discipline 
referral rates in a high school trial (Flannery, Fenning, Kato, & McIntosh, 2014). These studies 
provide important evidence on the links between SWPBIS implementation fidelity and valued 
outcomes and indicate the need to examine the fidelity process of change. 

Stages of Implementation 

Implementation science provides a useful lens for understanding school functions that 
mediate successful use of evidence based programs by providing a framework to understand 
implementation processes in complex organizational ecosystems like schools (Odom, 2009). For 
example, research on a statewide SWPBIS initiative documented practical, organizational 
barriers such as dedicated meeting time inhibited implementing SWPBIS with high fidelity 
(Kincaid, Childs, Blase, & Wallace, 2007). Work in other fields such as research on the 
Teaching-Family model for rehabilitation of youth offenders has shown by reviewing hundreds 
of attempted replications that implementation variables related to treatment fidelity were related 
to effective program use (Fixsen, Blase, Timbers, & Wolf, 2001). 

In a comprehensive synthesis of implementation research, Fixsen and colleagues (2005) 
provided an implementation framework that operationalized organizational capacities necessary 
for continued use of a program, and the stages of implementation. The stages of implementation 
involve six distinct phases of the implementation process highlighting that it “will not happen all 
at once or proceed smoothly, at least not at first” (Fixsen et al., 2005, p. 15). The six stages of 
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implementation provide research-based operationalized steps to aid the complicated process. The 
phases include an exploration and adoption phase where organizations evaluate their needs and 
find a program to meet their needs to full operation where the adopted program is being used 
with high levels of fidelity to innovation and sustainability phases where a program is adapted 
and maintained as part of day to day practice. 

Two important phases that occur between the exploration and adoption phase and the full 
operation phase are the program installation and initial implementation phases. Installation 
involves the use of organizational resources to prepare before the program is used with 
clients/students. The activities during this phase for SWPBIS involve focusing on 
implementation drivers related to successful use such as providing training for data use (e.g., 
Todd et al., 2011). Further, resources need to be directed towards the creation of leadership 
teams comprised of administrators and teachers, self-assessments of current behavior norms and 
outcomes must be completed, and school-wide behavior expectations must be established. The 
initial implementation phase involves using a program with students in school settings for the 
first time with varied levels of success. 

Fidelity Research 

Science is moving beyond a simple binary (i.e., implementing or not implementing) view 
of fidelity to a dynamic one. This is in part due to documented fidelity decreases when moving 
interventions from laboratory to school settings and the mediation between fidelity and valued 
outcomes (Crawford, Carpenter, Wilson, Schmeister, & McDonald, 2012; Hulleman & Cordray, 
2009; O’Donnell, 2008). Also, research has shown that fidelity is related to multiple school and 
teacher factors, including school size, teacher education, class size, and engagement during 
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training and coaching (Reinke, Herman, Stormont, Newcomer, & David, 2013; Zvoch, 2009, 

2012 ). 

Little research has examined fidelity change and factors influencing it. Flannery and 
colleagues (2014) showed the relation of SWPBIS fidelity to valued outcomes using growth 
methodology. In this research, the authors analyzed a discipline outcome using a latent growth 
model, and yearly fidelity scores were predictors of yearly office discipline referral rates. These 
results highlight that yearly fidelity levels relate to other longitudinal outcome changes, 
suggesting a hypothesis that fidelity change as a school-level outcome can be understood using 
growth methodology. Further, this study was conducted over three years and examined fidelity 
levels at the end of each year. Clearly, implementation fidelity also changes within each year as 
schools implement programmatic components as time and resources permit. Within-year fidelity 
growth can provide a valuable understanding of the implementation process. 

To the authors’ knowledge, only one study has focused explicitly on the longitudinal 
process of implementation fidelity. In a randomized trial of a class-wide peer tutoring program, 
Buzhardt, Greenwood, Abbot and Tapia (2006) developed a rate of implementation scale that 
measured the amount of time it took teachers to implement all of the intervention’s components. 
Results showed that (a) the measure accurately and objectively measured rate of completing 
tasks necessary for implementation, (b) schools varied widely in their implementation rates, and 
(c) the metric allowed documentation of implementation barriers. These results suggested that a 
fidelity growth metric can be a useful tool for researchers and practitioners to document 
organizational change processes during installation and initial implementation. 


Purpose of the Study 
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The goal of this study is to add to the empirical literature related to within-year fidelity 
growth during the installation and initial implementation of SWPBIS by exploring two research 
questions. First, to what extent does a repeated measure of SWPBIS fidelity document within- 
year fidelity growth within and between schools? Second, if the fidelity growth phenomenon 
exists as hypothesized, to what extent do contextual covariates used as typical controls in 
intervention trials (e.g., school size, relative socio-economic status) predict this growth? 

Method 

Participants 

Participants included 353 SWPBIS school teams comprised of building leaders and 
district coaches from schools throughout the United States. School characteristics were collected 
by the Institute for Education Science (IES) during the 2009-10 school year (National Center for 
Education Statistics, 2011). The majority of schools were in California (n = 46), Minnesota (n = 
116), Missouri (n = 107), and Wisconsin (n = 77). The majority of schools sampled were 
elementary schools (n = 223) and Title I eligible (n = 242). Ten schools were magnet schools, 
and five were charters. The smallest school enrolled 13 students, whereas the largest school 
enrolled 2,218 students. 

Measures 

Team Implementation Checklist (TIC). TIC Version 3.0 and 3.1 measures fidelity of 
SWPBIS during initial program implementation (Sugai, Homer, & Lewis-Palmer, 2009; Sugai, 
Homer, Lewis-Palmer, & Rossetto Dickey, 2011). As a fidelity measure, TIC measures 
adherence to the critical features of SWPBIS during the first few years of implementation. It is 
intended to be administered repeatedly (e.g., quarterly) throughout the first few years of SWPBIS 
to guide teams in the basic activities of initial implementation. SWPBIS teams and district 
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coaches self-administer the measure during SWPBIS team meetings to monitor and adapt formal 
implementation action plans. Research on TIC version 3.0 documented a high internal 
consistency (Cronbach a = .91), and concurrent validity (r = .59) with a validated SWPBIS 
fidelity and effectiveness evaluation measure, the Benchmarks of Quality (Tobin, Vincent, 
Homer, Rossetto Dickey, & May, 2012). Research on TIC version 3.1 resulted in high internal 
consistency ( a= .93) for an overall measure of SWPBIS implementation (McIntosh, Mercer, 
Nese, Strickland-Cohen, & Hoselton, 2014). Scale scores were formed by adding the 22 
consistent items representing the essential steps for initial and on-going implementation of 
SWPBIS that were reported on a three-point ordinal scale. Points were assigned to each position 
on the scale, with zero indicating not yet started, one for in progress responses, and two for 
achieved, with the highest score possible being 44. 

School context variables. School context variables included years implementing 
SWPBIS, locality, school size, and relative socio-economic status. The number of years 
implementing was obtained from a SWPBIS database. Locality, school size, and relative socio¬ 
economic status were contextual variables for each school that were gathered directly or 
converting from the 2009-10 school census (National Center for Education Statistics, 2011). The 
number of years implementing SWPBIS was defined as the number of complete school years the 
school has been actively completing TICs before the measurement occasion. This variable was 
coded as one for schools in their first year of implementation (i.e., the year they completed their 
first TIC), two for schools that have completed one full year of implementation and were in their 
second year, and so on. Locality was truncated from twelve original categories to four defined as 
city, suburb, town, and rural, and then converted to a series of dummy variables with city as the 
reference. School size was defined as a binary indicator with a zero representing schools with 
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fewer than or equal to 500 students and a one otherwise. Relative socio-economic status (SES) 
was measured via proxy calculated by dividing the number of students eligible for free and 
reduced priced lunch (FRL) and the total number of students in the school. 

Procedure 

This study used secondary data collected between 2009-2013 by a national SWPBIS 
technical assistance center (i.e., the OSEP Technical Assistance Center on: Positive Behavioral 
Interventions and Supports, www.pbis.org) and two previous studies (McIntosh et al., 2013; 
Mercer, McIntosh, Strickland-Cohen, & Homer, 2014). School SWPBIS teams completed the 
outcome measure, the Team Implementation Checklist (TIC) during team meetings and entered 
the results into an online database. Schools were included if in any given school year they (a) 
completed two or more administrations of the outcome measure, the TIC; and (b) were in years 
one to four of SWPBIS implementation. This process resulted in 500 unique school and school 
year combinations to explore within-year fidelity growth. 

Analytic Approach 

All models were estimated using the lme4 package (Bates, Maechler, & Bolker, 2012) for 
the R program (R Core Team, 2012). Time was coded as months from a zero point that aligns 
with the beginning of a school year. Subsequently, a school’s average slope represents the 
average fidelity change per month of school centered at the beginning of the school year. A 
multilevel growth model with varying intercepts and slopes was specified with the level-1 
equation: 

ytj = Koj + mj*time t + e tj , (1) 

where y t j equals the TIC score for school j at time t, noj is the initial status that varies between 
schools (i.e., the intercept), mj is the rate of change that varies between schools (i.e., slope), time t 
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is the time variable for each observation coded as months from the beginning of the school year, 
and e tJ is the random effect for school j at time t. The level-2 model was: 


7l0j = XB() + U0j 

( 2 ) 

7tij = XBi + U]j. 

( 3 ) 


X represents a matrix of variables hypothesized to predict fidelity growth with a leading column 
of ones to facilitate the estimation of an intercept. Bo represents a vector of fixed effect 
parameters predicting the growth model intercept, including the average fidelity score at time 
zero and average effects of years implementing and contextual covariates. Bi represents a vector 
of fixed effect parameters predicting the growth model slope, Kij, including an intercept and 
average effects for hypothesized predictors. Finally, uoj and uij represent the growth parameter 
random effects for each school. The between-school variance/covariance was specified 
according to Tao matrix specified in Equation 4 and is assumed to be independent from the level- 
1 residual, e t f. 


va r(u 0j ) 

_ co\(uoj,uij) var( u //)_' 


( 4 ) 


Results 


Preliminary Analysis 

A descriptive analysis was conducted to describe the outcome and predictors (see Table 
1). A close look at the time variable shows that on average schools completed TICs about five 
months into the school year, and observations were conducted at various points throughout the 
year. For the following descriptions, each unique school and school year combination will be 
described as schools. Given the exploratory nature of this study, schools provided varying 
numbers of waves of data with 291 schools having two waves, 140 schools having three, 64 
having four, three schools having five, one school having six and one school having seven waves 
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of data. Most schools were in Year 2 of implementation (j = 181). The remaining schools were 
in Year 1 (/ = 117), Year 3 (j = 143) and Year 4 (j = 57). Schools were located in all locality 
regions, with rural (j = 159) areas being the most represented and towns (j = 82) the least. The 
remainder of schools were roughly equally spread between cities (j = 136) and suburbs (j = 123). 
Slightly less than half of schools had more than 500 students (j = 215), ranged in proportions of 
FRL eligible students with about half of students eligible. A minority of schools were middle (j 
= 58) and high (j = 32) schools. 

Analysis 

Table 2 presents results of multilevel analyses to address the two main research 
objectives of this study. The middle columns documents results for an unconditional growth 
model used to assess the extent to which within-year fidelity growth can be analyzed with 
multilevel growth models. The second analysis highlighted in the rightmost columns shows 
results from of a model that added contextual covariates predicting fidelity growth. 

The first research question assessed the extent to which variability existed within and 
between schools, basically determining if analyzing fidelity growth using a multilevel growth 
model was appropriate. A model building approach utilizing change in deviance testing at each 
step concluded that the model outlined in Equations 1 through 4 was appropriate. To this end, 
three nested models without between school predictors were compared and deviance decreased at 
each stage: The first had a randomly varying intercept only (deviance = 9203.1), the second had 
a randomly varying intercept and fixed slope (compared to model one, x = 430.7, df= 1 ,p< 
.001), and the third, and interpreted model, had a randomly varying intercept and slope 
(compared to model two,/ 2 = 113.0, df= l, p < .001). Proportions of variance documented that 
18.29% of variance was within schools and 81.71% was between schools. Given these 
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proportionalities, it was concluded that fidelity growth was a varying phenomenon at both levels 
of analysis. 

The second research question builds on the first by assessing the extent to which between 
school contextual variables predicted within-year SWPBIS fidelity growth. In sum, 10 variables 
were entered into the analysis as predictors of both intercepts and slopes. Change in deviance 
tests showed that the inclusion of predictors significantly decreased deviance, indicating better 
model fit (% = 558.9, df= 20, p < .001). An additional model with non-significant predictors 
removed was estimated, resulting in a slight increase in deviance and a non-significant change in 
deviance test (% = 5.1, df= 11,/; = .93). Subsequently, the larger model with both significant 
and non-significant growth predictors was reported (Table 2), and significant results are 
interpreted. 

Intercept. Years implementing, a rural location, school size and being a high school all 
significantly predicted the average beginning of the year fidelity status (intercepts) across all 
schools. Table 2 shows the parameter estimates and standard errors, and Table 3 highlights 
average TIC scores and proportion of completed tasks for different types of schools based on 
significant effects. For each year implementing and controlling for all other variables, schools in 
cities, suburbs, and towns had equivalent average beginning year fidelity values, whereas rural 
schools scored three TIC points (or 10%) higher. High schools and large schools (i.e., more than 
500 students) had lower initial fidelity scores on average equating to roughly 6% lower than 
comparison schools. During Years 1 and 2, small schools located outside of rural areas had 
beginning of year fidelity scores of 20.82 (47% of total points) for elementary and middle 
schools and 18.13 (41%) for high schools. Large schools located outside of rural areas had 
slightly lower scores in Years 1 and 2 indicating a completion rate of 42% for elementary and 
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middle schools and 36% for high schools of fidelity tasks. Year 1 results indicated that schools 
made fidelity progress prior to the first full year of implementation, implicitly suggesting a 
hypothesis that many implementation tasks were completed in the previous school year or during 
the summer months (i.e., in preparation for initial implementation). Interestingly, schools in 
Year 2 of implementation (i.e., who had completed a full year of implementation) had roughly 
the same fidelity score at the beginning of the school year as schools in Year 1. As could 
logically be expected, schools in Years 3 and 4 had on average higher fidelity scores at the 
beginning of their school years. Except for large high schools outside of rural areas, schools on 
average completed more than 80% of fidelity tasks (indicating implementation of SWPBIS at 
adequate fidelity) by the beginning of implementation Year 4. 

Slope. Fidelity growth, documented here as the fidelity change per month of school, was 
significantly predicted by the year implementing and the proportion of students eligible for FRL. 
Schools gained 1.2 fidelity points per month of school on average and the rate varied most 
dramatically as a function of year implementing. No matter type or location, schools with an 
average SES (i.e., 48.7% of students eligible for FRL) gained 1.2 points per month during Year 
1, 1.68 points during Year 2, 0.72 during Year 3, and 0.52 during Year 4. Put another way, per 
month of school: (a) Schools in Year 1 completed about half a fidelity task, (b)Year 2 schools 
completed a bit less than one fidelity task, and (c) schools in Years 3 and 4 completed much less 
than a half of a fidelity task. These results indicated that schools in Year 2 of implementation 
gained momentum and implemented more critical features per month of school than in Year 1. 
Further, fidelity growth slowed during Years 3 and 4. Given the high initial fidelity scores for 
these schools, decreased growth rates were logical because schools that are further along in 
implementation approach the ceiling of the fidelity measure. Finally, schools with a higher 
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proportion of students eligible for FRL had lower fidelity growth rates. To provide an 
interpretation of this variable, a 10% increase in the percentage of students eligible for FRL 
above the mean related to roughly a 0.06 decrease in fidelity growth per month of school. 

Figure 1 graphically portrays the average numerical results for several significant factors 
in relation to the reference group of schools and controlling for other predictors. On the left, the 
varying intercepts and slopes of years implementing is shown with steeper slopes and lower 
intercepts for Years 1 and 2 and higher intercepts and less steep slopes for Years 3 and 4. As can 
be seen in the middle frame, small and large schools had different initial fidelity scores, but 
similar average within-year change. Elementary and middle schools had equivalent average 
growth patterns and high schools had lower intercepts and parallel slopes. 

Discussion 

This study showed that within-year fidelity growth, an implementation science construct, 
could be documented using a repeated SWPBIS fidelity measure, the TIC, and multilevel growth 
modeling methods. Building on this finding, school-level variables, including years 
implementing, location, and enrollment were found to be significant predictors of beginning of 
year fidelity scores. Further, years implementing and relative SES were significant predictors of 
growth rates. These results provide empiricism related to implementing evidence based 
programs at scale. 

Rate of fidelity growth was evaluated using multilevel linear models to account for data 
dependencies associated with repeated measures and provide insight into the proportion of 
variance within and between schools. This design was a unique use of these methods as many 
uses of multilevel growth models are used in the context of student-level phenomenon (e.g., 
McIntosh, Sadler, & Brown, 2012), whereas this study used multilevel growth models to 
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examine a school-level phenomenon. Findings add to implementation empiricism by building on 
previous work conducted by Buzhardt and colleagues (2006) on implementation rates through 
the use of a reliable fidelity measure and aforementioned growth methodology to document the 
within-year fidelity growth phenomenon. 

Interestingly, the strongest period of implementation growth was in Year 2 of 
implementation. The average initial fidelity estimate for Year 1 and Year 2 were statistically 
equivalent, whereas the growth rate significantly increased during Year 2. During the first year 
of implementation, schools completed approximately half of one program task per month, and 
during the second year of implementation, schools completed approximately one full 
implementation task. This result aligns with previous work on SWPBIS implementation in high 
schools that showed that significant advances in implementation outcomes were not realized 
until after the second year of implementation (Flannery, Frank, Kato, Doren, & Fenning, 2013). 

This study also adds to the growing field of fidelity research. First, it aligns with the 
theory that fidelity of implementation is dynamic and should be assessed at multiple points in 
time, especially early in the implementation process (Cross et al., 2015). Second, by showing the 
fidelity growth is mediated by multiple school-level factors supports a hypothesis suggested by 
Zvoch (2012) that fidelity is multidimensional. Second, given that schools in Years 1 and 2 had 
similar average initial fidelity estimates suggests that schools complete fidelity tasks prior to 
their first full year of implementation, perhaps in preparation for organizational change during 
the program installation phase. Although this pattern most likely helps dissipate some of the 
burdens in implementing new programs, it still took schools in this study upwards of two years 
on average to reach fidelity rates associated with full implementation. Finally, this study 
provides evidence on contextual factors affecting fidelity growth. 
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The contextual factors that affected fidelity growth included size, type, and relative SES. 
School size was a significant predictor of initial status but not growth rate, showing that large 
schools implemented with lower initial fidelity, but the rate of implementation was not inhibited. 
Conversely, relative SES was a significant predictor of growth rate but not initial status, 
suggesting that schools with students with more pressing economic needs have lower than 
average within-year SWPBIS growth rates. However, the magnitude of differences was 
relatively minor. Compared to elementary and middle schools, high schools had lower initial 
fidelity but equivalent growth rates. Like size, this finding suggested that high school 
implementation was more difficult than other settings because high schools had fewer critical 
features in place prior to SWPBIS adoption (Bohanon, Flannery, Malloy, & Fenning, 2009). 
Limitations 

Several limitations to this study must be mentioned. First, this study was exploratory, 
and causal inferences cannot be drawn. Second, the multilevel modeling techniques utilized here 
use regression methods that implicitly assume perfect measurement. Third, the outcome 
measures were self-evaluations, and external measures result in more accurate measurement 
(McIntosh et al., 2015). Fourth, the majority of schools in this study were elementary schools 
with less than 20% of the sample being middle and high schools, consequentially results may not 
generalize as accurately to middle and high school populations. Despite these limitations, this 
study provides valuable insight into the implementation process. 

Implications for Research and Practice 

Future research should continue to explore within-year implementation fidelity growth in 
relation to student outcomes. A logical direction would be to examine the relation of 
organizational systems change to change in student outcomes using Bronfenbrenner’s (1979) 
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ecological systems theory. Within this theory, meso-systems such as schools continually interact 
with children during development phases. Increases in SWPBIS fidelity have been previously 
shown to be associated with decreases in problem behaviors and improvements in student 
achievement outcomes, even though the approach does not focus directly on academic 
instruction (Homer et al., 2009; Luiselli, Putnam, Handler, & Feinberg, 2005), suggesting that 
improving the social settings of students provides more opportunity for instruction and therefore 
academic success. Future work should explore this finding using developmental theories to aid 
in explaining why programs focused on school functioning and student behavior mediate valued 
outcomes. 

The findings also support practice recommendations for educational leaders 
implementing SWPBIS. School, district and state leadership teams can compare their within- 
year fidelity growth rates to average rates reported in Table 3 to compare their progress to a large 
national sample. Leaders should also expect slower fidelity growth rates (i.e., lower slopes) in 
the first year of implementing than the second. Concern and additional support should be given 
to schools who have not reached an adequate implementation criterion by the end of the third 
year of implementation (i.e., achieved a fidelity percentage above 80%). Finally, high schools 
may need additional support to establish foundational SWPBIS features (e.g., establishing a 
representative school team that meets regularly) as these schools have lower beginning year 
fidelity scores on average. 

This examination of fidelity growth provided empirical documentation of the fidelity 
change process occurring during the installation and initial implementation stages (Fixsen et al., 
2005). Once deciding to adopt SWPBIS, schools incorporate the essential components for 
programmatic effectiveness that are reflecting in the items on the TIC measure. During these 
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phases, school teams, administrators and teachers must face multiple adaptive challenges to 
ensure effectiveness ranging from tangible challenges, such as creating a uniform office referral 
system to pedagogical alignment with behavioral principles implicit to SWPBIS. The TIC is a 
useful tool during these phases because it not only allows documentation of fidelity change, but 
is also designed to be used to guide revisions to implementation action plans that may benefit 
from knowing average fidelity scores at various points in the school year. Future research should 
explicitly examine multiple facets of fidelity growth during the distinct implementation stages. 
Although not tested, results from this study suggest that growth in more procedural steps of 
initial implementation (e.g., forming a team, collecting and assessing discipline data), is higher 
during installation than initial implementation. However, more substantive changes (e.g., 
changing the quality of classroom instruction; O’Donnell, 2008), might have a distinctly opposite 
growth pattern because of adaptive challenges mentioned and have slower growth during the 
earlier implementation phases and increase towards the latter. 

Conclusion 

In sum, the fidelity of SWPBIS use provides an excellent opportunity to examine 
implementation science phenomenon. Although this field is emerging, it is much needed to 
understand the factors promoting and inhibiting effectiveness of evidence-based practices in the 
field and at the school-level. Noted school improvement scholar John I. Goodland said when 
interviewed about school reform, “it doesn’t matter how many bills you pass and how many 
policies you lay down from on high - when it comes right down to it, the individual school has 
an incredible capacity for rejecting it passively or taking it on and doing something about it” 
(Novak, 1993). Implementation research provides an understanding for how schools can actively 


and effectively improve. 



SWPBIS FIDELITY GROWTH 


19 


References 

Algozzine, K., & Algozzine, B. (2007). Classroom instructional ecology and school-wide 
positive behavior support. Journal of Applied School Psychology, 24, 29-47. 
http://doi.org/10.1300/J370v24n01_02 

Bates, D., Maechler, M., & Bolker, B. (2012). I me 4: Linear mixed-effect models using S4 
classes. Retrieved from http://CRAN.R-project.org/package=lme4 

Bohanon, H., Flannery, K. B., Malloy, J., & Fenning, P. (2009). Utilizing positive behavior 

supports in high school settings to improve school completion rates for students with high 
incidence conditions. Exceptionality, 17, 30-44. 
http://doi.org/10.1080/09362830802590193 

Bradshaw, C. P., Koth, C. W., Bevans, K. B., Ialongo, N., & Leaf, P. J. (2008). The impact of 
school-wide positive behavioral interventions and supports (PBIS) on the organizational 
health of elementary schools. School Psychology Quarterly, 23, 462-473. 
http://doi.org/10.1037/a0012883 

Bradshaw, C. P., Mitchell, M. M., & Leaf, P. J. (2010). Examining the effects of schoolwide 
positive behavioral interventions and supports on student outcomes: Results from a 
randomized controlled effectiveness trial in elementary schools. Journal of Positive 
Behavior Interventions, 12, 133-148. http://doi.org/10.1177/1098300709334798 

Bronfenbrenner, U. (1979). The ecology of human development: Experiments by nature and 
design. Cambridge, MA: Harvard University Press. 

Buzhardt, J., Greenwood, C. R., Abbott, M., & Tapia, Y. (2006). Research on scaling up 

evidence-based instructional practice: Developing a sensitive measure of the rate of 



SWPBIS FIDELITY GROWTH 


20 


implementation. Educational Technology Research and Development, 54, 467-492. 
http://doi.org/10.1007/sll423-006-0129-5 

Crawford, L., Carpenter, D. M., II, Wilson, M. T., Schmeister, M., & McDonald, M. (2012). 
Testing the relation between fidelity of implementation and student outcomes in math. 
Assessment for Effective Intervention, 37, 224-235. 

Fixsen, D. L., Blase, K. A., Timbers, G. D., & Wolf, M. M. (2001). Offender rehabilitation in 
practice: Implementing and evaluating effective programs. In G. A. Bernfeld, D. P. 
Farrington, & A. W. Leschied (Eds.), Wiley series in forensic clinical psychology, (pp. 
149-166). New York, NY, US: John Wiley & Sons Ltd. 

Fixsen, D. L., Naoom, S. F., Blase, K. A., Friedman, R. M., & Wallace, F. (2005). 

Implementation research: A synthesis of the literature. Florida: University of South 
Florida, Louis de la Parte Florida Mental Health Institute, The National Implementation 
Research Network (FMHI Publication #231). 

Flannery, K. B., Fenning, P., Kato, M. M., & McIntosh, K. (2014). Effects of School-Wide 
Positive Behavioral Interventions and Supports and fidelity of implementation on 
problem behavior in high schools. School Psychology Quarterly, 29, 111-124. 
http ://doi. org /10.1037/spq0000039 

Flannery, K. B., Frank, J. L., Kato, M. M., Doren, B., & Fenning, P. (2013). Implementing 
Schoolwide Positive Behavior Support in high school settings: Analysis of eight high 
schools. The High School Journal, 96, 267-282. 

Homer, R. H. (2014). Compression implementation and scaling PBIS. Presented at the 


Wisconsin PBIS Network Conference, Wisconsin Dells, WI. 



SWPBIS FIDELITY GROWTH 


21 


Homer, R. H., Sugai, G., & Anderson, C. M. (2010). Examining the evidence base for School- 
Wide Positive Behavior Support. Focus on Exceptional Children , 42(8), 1-14. 

Homer, R. H., Sugai, G., Smolkowski, K., Eber, L., Nakasato, J., Todd, A. W., & Esperanza, J. 
(2009). A randomized, wait-list controlled effectiveness trial assessing school-wide 
positive behavior support in elementary schools. Journal of Positive Behavior 
Interventions, 11, 133-144. http://doi.org/10.1177/1098300709332067 
Hulleman, C. S., & Cordray, D. S. (2009). Moving from the lab to the field: The role of fidelity 
and achieved relative intervention strength. Journal of Research on Educational 
Effectiveness, 2, 88-110. http://doi.org/10.1080/19345740802539325 
Kincaid, D., Childs, K., Blase, K. A., & Wallace, F. (2007). Identifying barriers and facilitators 
in implementing schoolwide positive behavior support. Journal of Positive Behavior 
Interventions, 9, 174-184. http://doi.Org/10.1177/10983007070090030501 
Luiselli, J. K., Putnam, R. F., Handler, M. W., & Feinberg, A. B. (2005). Whole-School Positive 
Behaviour Support: Effects on student discipline problems and academic performance. 
Educational Psychology, 25, 183-198. http://doi.org/10.1080/0144341042000301265 
McIntosh, K., Massar, M., Algozzine, R. F., George, H. P., Horner, R. H., Lewis, T. J., & Swain- 
Broadway, J. (2015). Technical adequacy of the SWPBIS tiered fidelity inventory. 
Manuscript Submitted for Publication. 

McIntosh, K., Mercer, S. H., Hume, A. E., Frank, J. L., Turri, M. G., & Mathews, S. (2013). 
Factors related to sustained implementation of Schoolwide Positive Behavior Support. 
Exceptional Children, 79, 293-311. 



SWPBIS FIDELITY GROWTH 


22 


McIntosh, K., Mercer, S. H., Nese, R. N. T., Strickland-Cohen, M. K., & Hoselton, R. (2014). 

Predictors of sustained implementation of School-wide PBIS at 3 and 5 Years after Initial 
Implementation. Manuscript Submitted for Publication. 

McIntosh, K., Sadler, C., & Brown, J. A. (2012). Kindergarten reading skill level and change as 
risk factors for chronic problem behavior. Journal of Positive Behavior Interventions, 14, 
17-28. http://doi.org/10.1177/1098300711403153 

Mercer, S. H., McIntosh, K., Strickland-Cohen, M. K., & Horner, R. H. (2014). Measurement 
invariance of an instrument assessing sustainability of school-based universal behavior 
practices. School Psychology Quarterly, 29, 125-137. http://doi.org/10.1037/spq0000054 

National Center for Education Statistics. (2011). Common core of data: Public 

elementary/secondary school universe survey 2009-10 [Version la]. Retrieved from 
http://nces.ed.gov/ccd/ 

Novak, C. (1993). Interview with John I. Goodland. Technos Quarterly, 2(3). Retrieved from 
w w w. ait. net/techno s/tq_02/3 goodlad .php 

Odom, S. L. (2009). The tie that binds evidence-based practice, implementation science, and 
outcomes for children. Topics in Early Childhood Special Education, 29, 53-61. 

O’Donnell, C. L. (2008). Defining, conceptualizing, and measuring fidelity of implementation 
and its relationship to outcomes in K-12 curriculum intervention research. Review of 
Educational Research, 78, 33-84. 

R Core Team. (2012). R: A language and environment for statistical computing. Austria: R 
Foundation for Statistical Computing. Retrieved from http://www.R-project.org/ 

Reinke, W. M., Herman, K. C., Stormont, M., Newcomer, L., & David, K. (2013). Illustrating 
the multiple facets and levels of fidelity of implementation to a teacher classroom 



SWPBIS FIDELITY GROWTH 


23 


management intervention. Administration and Policy in Mental Health and Mental 
Health Services Research, 40, 494-506. http://doi.org/10.1007/sl0488-013-0496-2 
Sugai, G., Horner, R. H., & Lewis-Palmer, T. (2009). Team Implementation Checklist (v. 3.0). 

Eugene, OR: Educational and Community Supports. Retrieved from http://www.pbis.org 
Sugai, G., Horner, R. H., Lewis-Palmer, T., & Rossetto Dickey, C. (2011). Team Implementation 
Checklist (version 3.1). Eugene, OR: Educational and Community Supports. Retrieved 
from http://www.pbis.org 

Tobin, T. J., Vincent, C. G., Homer, R. H., Rossetto Dickey, C., & May, S. A. (2012). Fidelity 
measures to improve implementation of behavioural support. International Journal of 
Positive Behavioural Support, 2(2), 12-19. 

Todd, A. W., Horner, R. H., Newton, J. S., Algozzine, R. F., Algozzine, K. M., & Frank, J. L. 
(2011). Effects of team-initiated problem solving on decision making by schoolwide 
behavior support teams. Journal of Applied School Psychology, 27, 42-59. 
http://doi.org/10.1080/15377903.2011.540510 
Zvoch, K. (2009). Treatment fidelity in multisite evaluation: A multilevel longitudinal 

examination of provider adherence status and change. American Journal of Evaluation, 
30, 44-61. http://doi.org/10.1177/1098214008329523 
Zvoch, K. (2012). How does fidelity of implementation matter? Using multilevel models to 
detect relationships between participant outcomes and the delivery and receipt of 
treatment. American Journal of Evaluation, 33, 547-565. 
http://doi.org/10.1177/1098214012452715 



SWPBIS FIDELITY GROWTH 


24 


Table 1 

Descriptive Statistics 



n 

M 

SD 

Min 

Max 

TIC score 

1286 

30.08 (68%) 

9.33 (21%) 

0 

44 

Initial TIC score* 

132 

19.61 (45%) 

11.47 (26%) 

0 

42 

Time 

1286 

4.75 

2.94 

0 

10.97 

% FRL** 

498 

48.68 

23.21 

0 

98.07 


Note. TIC score M’s and SD’s indicate raw implementation fidelity scores. Proportion of fidelity 


task completion (i.e., TIC total percentage) is in parentheses. Time was coded as months from 
the beginning of the school year. *Descriptive statistics for initial TIC scores were from 
observations taken at time = 0 to time = 0.5. **Two schools were missing data on the number of 
students eligible for FRL. 
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Table 2 


Multilevel Growth Model Results for Within-Year Fidelity Growth 



Unconditional Model 

Conditional Model 


Est. SE 

Est. 

SE 

Fixed effects: Intercept 

24.98*** 0.52 

20.82*** 

1.36 

Year Imp 2 


-0.26 

1.22 

Year Imp 3 


10.72*** 

1.30 

Year Imp 4 


14.04*** 

1.68 

Suburb 


-0.16 

1.33 

Town 


0.55 

1.49 

Rural 


3.00** 

1.28 

>500 


-2.47** 

0.97 

Prop. FRL 


3.02 

2.10 

Middle 


0.55 

1.26 

High 


-2.69* 

1.56 

Fixed effects: Slope 

1.17*** 0.06 

1.20*** 

0.17 

Year Imp 2 


0.48** 

0.16 

Year Imp 3 


-0.68*** 

0.17 

Year Imp 4 


-0.80** 

0.22 

Suburb 


0.06 

0.16 

Town 


0.11 

0.19 

Rural 


-0.09 

0.16 

>500 


0.06 

0.12 

Prop. FRL 


-0.55* 

0.25 

Middle 


0.00 

0.16 

High 


0.03 

0.20 

Random effects 




Level 1: var (e t] ) 

21.57 

20.33 


Level 2: var(«o/j 

96.27 

60.28 


Level 2: var(w/ 7 ) 

0.06 

0.35 


Cor (u 0 j, uij) 

Proportion of variance 

-0.87 

-0.83 


Level 1 

0.1829 

0.2511 


Level 2 

0.8171 

0.7489 



Note. The models reported here are for 1286 fidelity observations for 500 unique school/ 


school year combinations. *p < .05. **p < .01. ***p < .001. 
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Table 3 

Average Within-Year Fidelity Growth Estimates and Proportion of Fidelity Task Completion 
for Statistically Significant Effects Described by Implementation Year 


Elementary & Middle Schools High Schools 



Year 

Year 

Year 

Year 

Year 

Year 

Year 

Year 


1 

2 

3 

4 

1 

2 

3 

4 

Beginning year fidelity status (intercept) 






Rural 









Small 

23.8 

23.8 

34.5 

40.9 

21.1 

21.1 

31.9 

38.2 

(54%) 

(54%) 

(79%) 

(93%) 

(48%) 

(48%) 

(72%) 

(87%) 

Large 

21.4 

21.4 

32.1 

38.4 

18.7 

18.7 

29.4 

35.7 

(49%) 

(49%) 

(73%) 

(87%) 

(42%) 

(42%) 

(67%) 

(81%) 

Large 

18.4 

18.4 

29.1 

35.4 

15.7 

15.7 

26.4 

32.7 

(42%) 

(42%) 

(66%) 

(80%) 

(36%) 

(36%) 

(60%) 

(74%) 

All other schools 

20.8 

20.8 

31.5 

37.9 

18.1 

18.1 

28.9 

35.2 

(47%) 

(47%) 

(72%) 

(86%) 

(41%) 

(41%) 

(66%) 

(80%) 

Fidelity growth per month (slope) 







Low SES* 

1.3 

1.8 

0.6 

0.5 

1.3 

1.8 

0.6 

0.5 

( 3 %) 

(4%) 

(1%) 

(1%) 

( 3 %) 

(4%) 

(%1) 

(1%) 

Ave. SES* 

1.2 

1.7 

0.5 

0.4 

1.2 

1.7 

0.5 

0.4 

( 3 %) 

(4%) 

(1%) 

(1%) 

( 3 %) 

(4%) 

(1%) 

(1%) 

High SES* 

1.1 

( 3 %) 

1.6 

(4%) 

0.4 

(1%) 

0.3 

(1%) 

1.1 

( 3 %) 

1.6 

(4%) 

0.4 

(1%) 

0.3 

(1%) 

All other schools 

1.2 

( 3 %) 

1.7 

(4%) 

0.5 

(1%) 

0.4 

(1%) 

1.2 

( 3 %) 

1.7 

(4%) 

0.5 

(1%) 

0.4 

(1%) 


Note. Numbers indicate raw TIC scores. Proportion of fidelity task completion (i.e., TIC total 


percentage) is in parentheses, and was calculated by dividing the average growth estimates by 


the total possible fidelity points (44). * Average SES denotes 48.7% of students eligible for 


FRL; low and high SES denotes -/+ 15% respectively. 
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Figure 1. Average conditional growth model estimates of four factor covariates when controlling 
for other predictors. 












